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ABSTRACT 

The Operator Control Station of the JPL/NASA Telerobot 
Demonstrator System provides the man-machine interface between 
the operator and the System. It provides all the hardware and 
software for accepting human input for the direct and indirect 
(supervised) manipulation of the robot arms and tools for tas 
execution. Hardware and software are also provided for the display 
and feedback of information and control data for the operators 
consumption and interaction with the task being executed. This 
paper, Part II, addresses the software design of the OCS. 


1.0 INTRODUCTION 

The JPL/NASA (Jet Propulsion Laboratory / National Aeronautics and Space 
Administration) Telerobot Demonstrator System is a research testbed for the 
development, integration and testing of advanced robot control techno ogies. . T e 
component technologies and system-wide design experiences derived from such a 
system development and technology demonstration are targeted for use in future space 
programs, including the NASA^s Flight Telerobot Servicer project. 

The Operator Control Station (OCS) is a part of this Telerobot Demonstrator 
System. It contains state-of-the-art hardware, both mechanical and computing for 
providing control input to the System. It contains software, in controls as well as 
human operator interface, for real-time and user-friendly interaction. Video dls P'^V® 
for text, graphics and camera images are provided for operator consumption, where 
appropriate, voice input/output is provided to reduce operator work-load . Data 
manipulation such as object designation capability is provided for efficient task 
definition and execution. Access to all Telerobot subsystems is provided for software 
development and on-line monitoring. 

The OCS system design and hardware configuration have been discussed in details 
in Part I of this paper [ref.lj. This paper, Part II, concentrates on the software design. 
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2.0 OCS SOFTWARE REQUIREMENTS 
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III. 


IV. 


V. 


VI- 


VII. 


VIII. 


IX 


X. 


Inputs from both workstations shall not be prioritized, and shall be processed 
sequentially by the time order that the inputs arrive; 

Flexibility shall be maintained in the overall structure, grouping, and individual 
specification of the set of commands; the entire set shall be kept ' n a separate 
command definition file, which can be modified and edited independent of t 

compilation of the main OCS software; .. cot 

Similar flexibility shall be maintained in the set of messages, the entire set 
shall be kept in a separate message definition file, which can be ed 

The' set ^"vocabulary words, grouping and grammar of the speech input sentonces 
shall be designed to achieve 98%+ Type I recognition accuarcy and 95/o+ Type II 
false alarm rejection accuracy; with the use of interrogation and operator 
interaction, the overall accuracy achievable shall be improved to 99 /«+, 

The accuracy of the object designation process shall be within one pixel RMS 
(root mean square) averaged over the vertices of the object; (absolute ^curacy 
is not specified, because it is a factor of the distance of the object from the 

cameras and depends on camera model accuracies); . nr e 

The primary OCS workstation mouse shall be shared between the normal OCS 
command process and the object designation process which utilizes the mouse 
for graphic entries; (this property eliminates the proliferation of mouse(s), eac 
graphic overlay machine for the object designation process requires one mouse). 
During the object designation process, OCS commands shall continue to 
receivable through voice input; keyboard and mouse inputs can be togg ed 
between the designation and command processes, using a switch on the mouse, 
Menus and direct keyboard inputs shall be designed to good human engmnering 

standards. 


3.0 OCS SOFTWARE DESIGN 


The OCS software consists of ten processes distributed among a Sun-3/160, 
which serves as the primary workstation, a Sun-3/60, the secondary workstation, and 
a DEC pVAX, which is the communications gateway computer for interface among the 
other subsystems. These processes provide the following OCS functions: 
o Command Interpreter 
o Message Processing and Display 
o Ethernet Interface, Sun/microVAX 
o External Subsystem Interface 
o Video Switch Operation and Control 
o Wire Frame Object Designation 


The dual workstation design allows OCS commands to be issued from either the 
primary or the secondary workstation, and provides for messages generated on either 
workstation, or by external subsystems, to be displayed on both workstation monitors. 
Voice input and audio output capabilities have been included to aid the operator. A 
continuous speech recognizer is provided to accept operator voice commands, which 
are routed to the OCS Command Interpreter. A speech synthesizer is provided to 
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acknowledge voice commands, as well as annunciate important messages that may 
occur when the operator's attention is directed away from the workstation display. 
Multiple voices are utilized to assist the operator in message recognition. 

The processes which constitute the OCS program set are illustrated in Figure 1. 
The SunView Notifier is shown with the processes that provide direct window 
interface to the users by either accepting operator input or displaying messages. 
Inter-process communication methods used by the OCS software, include Remote 
Procedure Calls, Unix Sockets, Pipes, and DECnet. Figure 1 also shows the Ethernet 
dividing the primary from the secondary workstation, and separating the OCS software 
which resides on the Sun workstations from the pVAX gateway computer and external 
subsystems. 

The OCS Command Interpreter resides on both the primary and the secondary 
workstation, and receives operator commands from the speech recognizer, as well as 
the standard Sun keyboard and mouse. Interpreted command data is sent to other OCS 
functions in the form of command interface records by means of Sun's Remote 
Procedure Call (RPC) and external Data Representation (XDR) facilities, which provide 
for inter-process communication regardless of the host in which the serving process 
resides. 

The OCS command interface, in addition to allowing direct keyboard and voice 
commands, provides a customizable multi-level menu interface that is controlled by 
the Sun three-button mouse. Menu display is activated by clicking a mouse button, or 
selecting an icon, and command selections are made by positioning the mouse cursor 
over the desired option. Many menu items contain a "pullright" capability, which, when 
selected by the mouse cursor, causes the next level of menu options to be displayed to 
the right of the current selection. The operator may specify, during OCS activation, 
the degree to which previous menu selections influence subsequent menu display. 

The Message Processor accepts message definition records, which contain 
message ID’s and message attributes, from all OCS processes, and generates messages 
for display on the Sun monitors and for annunciation by means of the DECtalk speech 
synthesizer. Messages targeted for monitor display are routed to the message display 
software on both the primary and the secondary workstations so that all messages are 
displayed on each workstation. 

The Ethernet Interface function of the OCS provides the Sun interface with the 
pVAX gateway computer. This function accepts command interface records from 
processes on either Sun workstation, extracts pertinent data for interface with 
external subsystems, and sends host-to-gateway records over the Ethernet to the 
External Subsystem Interface function on the pVAX via DECnet. The Ethernet Interface 
function also accepts gateway-to-host records over the Ethernet from the External 
Subsystem Interface function, extracts pertinent data, and distributes the data to the 
appropriate OCS process by means of (1) command interface records using RPC 
facilities, (2) message definition records, also using RPC, and (3) data written on a 
Unix socket connected with the Video Switch Operation function. 
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The External Subsystem Interface function resides on the pVAX as the 
communication interface between the OCS software that executes on the Sun 
workstations and the external subsystems. This inter-subsystem communication is 
done using Network Service Transactions (NSTs) which are controlled by the NIP. The 
External Subsystem Interface function receives host-to-gateway records over the 
Ethernet from the primary Sun host via DECnet services, and uses the data to initiate 
or respond to NSTs with other TDS subsystems. In addition, NST records that originate 
from external subsystems are received and validated by this function. Validated data 
is extracted from the NIP NSTs, formatted into gateway-to-host records, and sent via 
DECnet to the Ethernet Interface function on the Sun primary workstation. 


The Video Switch Operation and Control function resides on the primary Sun 
workstation and provides the software interface to the OCS video switch hardware. 
The Video Switch Operation function interfaces with the external S&P Subsystem, by 
means of the OCS Ethernet Interface and External Subsystem Interface functions, to 
request remote switching of the cameras and frame buffers to operator selected video 
channels, and to receive the remote video switch status. 


The Wire Frame Object Designator function resides on the primary Sun 
workstation and provides the capability for objects whose position and orientation are 
unknown or invalid to be designated and subsequently used in the System database. The 
designation is performed by means of wire frame diagrams, based on object and 
camera model data, being overlayed on actual video images, manipulated in terms of 
translation and rotation, and fit with the video image by means of vertex designation 
and a least-squares fit algorithm. The Wire Frame Object Designator function receives 
object and camera model data from the external TPR Subsystem, by means of the OCS 
Ethernet Interface and External Subsystem Interface functions, and returns updated 
camera model data after an object has been designated. In addition, object designation 
commands from the operator are received in command interface records sent by the 
OCS Command Interpreter process. 


An extensive user interface for the Teleoperation Subsystem is provided to 
facilitate both operator input and status recognition for the robot manipulators and 
the force-reflecting hand controllers, their states, joint limiting situations, operating 
modes, etc. Numerous icons are provided for the selection of the various teleoperation 
command and status windows. A direct communication link with the Teleoperation 
computer is provided, and voice teleoperation commands received by the Command 
Interpreter function are routed to the Teleoperation Subsystem by this link. 


The OCS software has been designed to operate within the SunView Windows user 
interface environment. The OCS window layout can be customized by the operator to a 
large degree, including the size and position of the windows, the default character 
font, icon positioning, the look of menus, etc. The windows may be moved about the 
Sun display, resized, or closed into icons, and many window option defaults may be 
selected according to the operator's desires. Easily identifiable iconic representations 
for each of the OCS windows have been provided, and the mouse cursor image is 
modified to indicate the active window in order to give additional visual cues to the 


operator. 
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Integrating the OCS operator interface with the SunView environment also 
provides a flexible capability for the operator to perform terminal emulation with all 
the external subsystems. Terminal emulation windows may be opened when direct 
interface with another subsystem is required, and subsequently destroyed, or left 
active in an iconic state for future use. Multiple terminal emulation windows may be 
active simultaneously, in addition to the OCS processes. 

This implementation approach allows the operator a great deal of flexibility in 
the user-interface with the OCS, and provides a familiar look and feel for a user 
experienced with the Sun window environment. 

The following three subsections further describe the Command Interpreter 
process, the Message Definition and Display process, the Ethernet and External 
Subsystem Interface process. Explicit discussion on the Video Switch Operation and 
Control is avoided because of the simplicity of the process. Discussion on the Object 
Designation process has been given in [ref. 1,4]. 

3.1 Command Interpreter 

Table 1 is a condensed table of the OCS command set. The OCS Command 
Interpreter function provides a flexible, table driven capability for command 
definition, recognition, interpretation, and inter-process distribution. Valid 
commands for direct keyboard entry, voice input, and menu selections, as well as 
command and command-qualifier relationships, are defined in a standard text file, the 
command definition file, and can be modified by the user to change the command 
interpretation and processing traits of the OCS without modification of the software. 

The command definition file completely defines the command processing 
characteristics of the OCS. Specific keyboard and voice commands are defined, as are 
command groupings, menu content, and the text of menu selections. Command 
translation data is defined for process specific or device specific information to be 
supplied in conjunction with a particular command. The command destination--that 
is, the target program that will process the command, is also specified for each 
command, as is any required NST data associated with the command, such as the 
destination subsystem, record type and identification, and subsystem interchange 
protocol. Each command may be specified as requiring confirmation before being 
processed, providing a measure of protection from inadvertent command selection. 

The purpose of the command definition file and the Command Interpreter 
flexibility is to allow the OCS and Telerobot Demonstrator System functionality to 
evolve, without the necessity of redesigning or modifying the command processing 
function. Additional processes which may be added to the OCS at a later date, will be 
able to use the same user interface, yielding consistency as the System capabilities 
evolve. Commands may be grouped into menus, and the text of the menus may be 
modified after the user has gained experience with the system and becomes aware of 
changing needs. In fact, multiple command definition files may be defined to allow 
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each operator to customize his or her interface with the OCS according to individual 
preference or the tasks to be performed. 


The operator interface to the Command Interpreter function makes use of the 
SunView Window services, and the SunView Notifier, which directs processing control 
based on user input by direct keyboard entry, mouse-based menu selections, or voice 
input. Voice input from the speech recognizer is directed to the Command Interpreter 
function, regardless of which SunView window is active. 


Once the Command Interpreter has received a command from the operator, the 
command is looked up in the command definition table, which is built at initialization 
from the command definition file. Commands and their associated qualifiers are 
validated, and command confirmation is requested when so indicated. Command 
validation or cancellation may also be performed by use of keyboard or voice input, or 
bv clicking the mouse buttons. For valid, confirmed commands, the data in the 
command definition table is used to build a command interface record, which is then 
sent to the specified destination for processing. 


The Sun RPC (Remote Procedure Call) services are used for inter-process 
communication between the Command Interpreter, which resides on both the primary 
and the secondary workstation, and the other OCS functions, which operate on the 
primary workstation. Together with the XDR (external Data Representation) services 
provided by Sun, which organize data bytes in a machine independent format, RPC 
allows an inter-process communication client process to communicate with a server 
process on any valid host computer. These facilities enable the Command Interpreter 
to execute on either workstation without requiring special-purpose software for 
network interface from the secondary workstation's Command Interpreter to OCS 
functions that reside on the primary workstation. 


3.2 Message Processing and Display 

Table 2 is a condensed table of the OCS message set. The OCS Message 
Processing and Display function provides a centralized, table driven interface to the 
OCS message windows on the Sun workstations, and the voice output device. 

The Message Processing server process for RPC from other OCS functions resides 
on the primary workstation as the target of all Message Definition Records. This 
process builds the message text based on the incoming data and the specification in 
the Message Definition Table, and routes the message text to the DECtalk speech 
synthesizer, and the Message Display software residing on both the primary and the 
secondary workstations. Audio output may be enabled or disabled by normal OCS 
commands, and the gender of the message voice is selected based on the attributes 
specified in the Message Definition Table. 

A standard ASCII format Message Definition File is used for the specification of 
message ID's, attributes, and text, which are processed into the Message Definition 
Table during initialization. The message attribute specification allows a message to 
be defined as a normal or critical text message, and/or a male voice, or female voice 
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message. The use of the Message Definition File allows messages to be added or 
changed, or the characteristics of a message to be revised based on evolving system 
needs, without requiring redesign or modification of the existing OCS software. 

A separate message window is utilized to inform the operator of critical 
messages. The display of a critical message is accompanied by an audible beep a 
visual flashing of the critical message window, and if the message window had been 
closed into an icon or hidden behind another window, it is automatically opened and 
exposed to operator view. Both critical and normal messages are displayed in the 
normal message window, which provides a context for any critical messages that may 
be generated during an operational session. 

All valid commands are acknowledged by messages in the normal message 
window, and all messages are displayed at both the primary and the secondary 
workstation with an indication of the workstation that generated the message. These 
features allow each OCS workstation operator to be aware of the actions of the other 
operator, as well as any system or error messages that arise, and provides a history of 
commands and how they were generated. 

The capability to store OCS messages from both the critical and normal message 
windows to a user specified file is provided, along with the option for the operator to 
clear each message window individually - to eliminate outdated critical messages, for 

example. The workstation mouse is used to activate a menu and select the desired 
option. 


The OCS message windows may be closed in icons or positioned on the 
workstation display independently from the other OCS windows, such as the command 
window. This flexibility allows the OCS message display to be rearranged according to 
individual operator preferences or processing needs. 

3.3 Ethernet and External Subsystem Interface 

The Ethernet Interface and External Subsystem Interface functions of the OCS 
work together to provide a centralized interface between the Sun workstations and the 
pVAX gateway computer, and between OCS and the external subsystems. 

The Ethernet Interface function consists of two separate processes, one for 
sending host-to-gateway records from the Sun to the pVAX, and the other for receiving 
gateway-to-host records from the pVAX. Upon initialization of the Ethernet Interface 
function on the Sun primary workstation, DECnet services are used to activate the 
External Subsystem Interface function, which executes on the communications 
gateway pVAX, and establish a communication link over the Ethernet between the two 
functions. 

The External Subsystem Interface function receives host-to-gateway records 
from the Sun, extracts the data and builds the NIP/NST records to be sent to the other 
subsystems, then interfaces with the NIP to ship the NST records. All interface with 
the NIP software is performed in this function, which also receives NIP/NST records 
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directed to the OCS from the other subsystems. Data is extracted from the incoming 
NST records, packaged into gateway-to-host records, and sent to the Ethernet 
Interface function over the Ethernet. 

The Ethernet Interface function receives the gateway-to-host records S0 nt from 
the External Subsystem Interface function, and examines the record content to 
determine which OCS process should receive the data. Message data is sent by means 
of RPC’s to the Message Processing function. Other data is used to generate Command 
Interface Records, which are also sent via RPC to the destination process Video 
switch status from the S&P subsystem is sent over a connected Unix socket to the 
Video Switch Operations and Control function, which is waiting for the status .in order 
to complete a video switch setting operation, while object and camera model data i sent 
by the TPR subsystem is distributed to the Wire Frame Object Designation function 

using an RPC interface. 

Current design contains the following NST's. (OCS-to-SE): (i) Conim_Lmk_Test, 
(ii) SE Start-Up, (iii) SE Shut_Down, (iv) SE_Halt. (OCS-to-S&P): (v) SelectVideo 
Sources, (vi) Read_Video_Sources. (OCS-to-TPR): (vii) V,s.on_Arm_Control, (v.n) 
Object_Designate. (OCS-to-all): (ix) Message. 


4.0 SUMMARY 

As mentioned in Part I of this paper, the present OCS design is an evolutionary 
design which will evolve and change as the telerobot technology matur f s - b0 ‘ h 
system design and in component design. The present design isbelievedtohavethe 
necessary ’hooks and scars' for future system expansion, both in software and in 
hardware. Despite its flexibility, certain architectural features are recognized to be 
suboptimal because of project constraints. These constraints have been discussed in 

Part I of this paper. 

After complete integration in the Telerobot Demonstrator System testbed 
laboratory in Summer, 1989, the OCS will serve as the focal point of the Demonstrator 
System. Real hands-on operational flow analysis and workload ana ys.s will be 
conducted, so as to evaluate the effectiveness of the OCS design, and of the integrated 
telerobot system. More research and development items, improvements on point- 
designs, alterations of physical layout, addition of vocabulary, etc. will undoubtedly 
surface when more experience is gained from OCS and telerobot experiments. As o 
powerful computers become available, and as the understanding of a telerobot system 
matures, the state-of-the-art OCS technology will evolve. 
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T©lbO© H* (condensed) ’KEYBOARD’ & ’’Voice" 


HOST CONTROL : 

VIDEO SWITCH CONTROL : 


ORIFCT DESIGNATION: 
VISION ARM CONTROL : 

TFIEOP CONTROL : 

SYSTEM EXEC COMMANDS: 


'Quit OCS’ ("Quit OCS’’); ’Audio’ ("Enable Audio"); 'Noaudio'; 

'Audio Ack’; 'Noaudio Ack’; 'Vsdflt'; ’Swst’ 

'Cl' ("CameraOne"); ’C2’; . . . ; ’C5’ 

'B1' ("BufferOne"); ’B2’; . . . ; ’B4’ 

'CHI’ (’’ChannelOne); ’CH2’; . . . ; ’CH5' 

'RFT' ("RightForceTorque"); ’LFT 

'STE' ("StereoCameras); 'OH' ("OverheadCameras") 

■RW' ( ’’RightWing"); ’LW’ 

’WF’ ("WireFrame"); 'DROP' ("Drop") 

'SWST' 'parameters' ("SwitchStatus" "parameters") 

•conj'; 'top'; 'bot'; 'left'; 'right'; ’rear'; 'rotate'; 'manip'; Tit’;’undo’; 
’erase’; ’done’; ’cancel’; ’next’, ’color'; 'wf_cont'; 'select'; 'clear 
'va' - 'vastop'; 'hold'; 'a' ("moveagain"), 'back ; goback , 

■wm' ("World Mode") 'tm ("ToolMode"); 'PI' ("Movetopositionl") 
•P2'; ...; f ; 'b'; Y; T; 'u'; 'd'; ’pr"; 'pi'; 1u'; 'td'; 'mm' ("move- 
more"); 'sa'; 'lm'; 'Is'; 'bm'; 'ss'; 

'teleop’; 'telestop’; 'idx'; 'lidx'; 'ridx'; 'pos'; 'Ipos'; 'rpos'; 

•rat’; ’Iraf; ’rraf; ’jnf; ’Ijnf; 'rjnf; ’crnt’; 'Icrnf; ’rcrnf; ’tst’; 

’Itst’; ’rtst’; ’one’; ’two’; ... 

’systart’; ’shutdown’; ’halt’; 


TaB&O® S. (condensed) ’Message window' & "Voice" 


MESSAGE DEFINITION FILE LISTS ALL MESSAGES BY: MESSAGE ID; ATTRIBUTE; AND TEXT. 
ATTRIBUTES ARE: (1) normal; (2) critical; (4) male voice; (8) female voice. 


INTFRNAL MESSAGES: 


VIDEO SWIT CH warning: 

WIREFRAME OB DES errors: 
WIREFRAME OB DES warnings: 

WIREFRAME OB DES status: 

VDT MONITOR warnings: 

VDT MONITOR Status 

SUBSYSTEM messages: 


'quitting'; 'initializing'; 'input error on %s'; 

'command %s is not recognized’ ("%s is not recognized''); 
'error - getting OCS video switch routing status %s'; .... 
'unknown video switch name %s* ("named video switch not 
known"); ’invalid video switch command , ... 

’bad camera # received’ ("requested selection not done ), ... 
‘unknown obj des command %s‘ ("illegal command ), 

'cannot fit, no points saved' ("no points saved"); 

'object designator active' ("object designator active"); 

'mouse being used for object designation'; 

"no point to erase!", "showing different camera view!";^ .... 
”%s points saved!"; "object rotated to show top view , 

"undo done!"; "object partially out of camera view"; ... 

•MCM requires attention' ("the MCM requires attention"); ...; 
'va limit stop joint 1'; ...;’SE status: warning (1)';.... 
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